AMENDMENTS TO THE SPECIFICATION 

On page 1, please replace the subheading "BACKGROUND OF THE 
INVENTION" with "BACKGROUND". 

On page 3, please replace the subheading "SUMMARY OF THE INVENTION" 
with "SUMMARY". 

On page 10, please replace the subheading "DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS" with "DETAILED DESCRIPTION OF THE VARIOUS 
EMBODIMENTS". 

Please replace Paragraphs [0001], [0009-0013], [0022], [0024-0027], [0029- 
0031], [0034-0035], [0037], [0047], [0071], [0073], [0099], [0103-0113], with the 
following paragraph rewritten in amendment format: 

[0001] The present i nv o nt i on disclosure is directed to image registration, and more 
particularly to a system and method for automatically registering images of different 
perspectives and images from sensors with different internal geometry. 

[0009] The match process of the present inv e nt i on disclosure solves the problem of 
registering images of different perspectives and images from sensors with different 
internal geometry. 

[0010] Generally, the present i nv e ntion disclosure addresses the problem of relating 
sensor images to ground coordinate systems with high accuracy. This is accomplished 
by registering or aligning the sensor image with a precision geocoded reference image. 
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Because of this high precision, the geocoding of the reference image can be transferred 
to the sensor image with accuracy comparable to that of the reference image. The 
geocoded reference image, such as a DPPDB image provided by the National Imagery 
and Mapping Agency, provides a known accuracy in relation to ground coordinates. The 
present Invefttjen disclosure also solves the problem of accurately registering a small 
sensor image to a much larger reference image, which may be taken as a stereo pair of 
images for some embodiments of this invontion disclosure where the two images have 
significantly different perspectives of the scene. 

[0011] One aspect of this invention disclosure makes use of knowledge of the 
approximate location of the scene as it is found in the reference image to limit the 
search area in attempting to match the small image to the larger image. Another aspect 
of the disclosure is the use of approximate knowledge of the sensor location and 
orientation, or the sensor model, at the time when the scene is imaged, as that 
knowledge, combined with knowledge of the scene location, may be used to reduce the 
search process. Yet another novel aspect is the use of the geometry of the scene area, 
as known or derivable for the reference image around the scene area, or as known or 
derivable for the sensor image, to modify one or both of the images to have a common 
geometry; that is, to eliminate perspective differences that arise from the two different 
views of the scene as imaged separately by the sensor and the reference. 
[0012] Further in accordance with the invontion disclosure , knowledge of the sensor 
location and orientation and of the location of the scene is used to extract a small 
portion or "chip" of the reference image or images that encompasses the scene area 
imaged by the sensor. 
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[0013] Parameters of the sensor, such as field of view and resolution, together with 
measurements of range and directions in three dimensions to the scene depicted in the 
sensor image, determine a nominal "sensor footprint", or prospective location, 
orientation and size for the sensed scene and for the reference chip. However, these 
measurements are actually estimates that involve uncertainties, producing uncertainty 
in where the sensed area or footprint actually is and in its actual orientation and size. It 
can be noted that these same uncertainties also produce or involve the fundamental 
inaccuracies that this i nv e nt i on disclosure is intended to overcome. The uncertainties 
are, however, known quantities, and are usually expressed in terms of error bounds on 
each measurement. This makes it possible to determine an uncertainty basket around 
the nominal sensor footprint, such that the scene's true location and its full extent will 
always fall within that uncertainty basket. The uncertainty basket defines the portion of 
the reference image to extract as the reference chip. 

[0022] Of particular importance is the ability obtained using the invont i on disclosure to 
identify the specific location in the reference image of a target point appearing in the 
sensor image, when said target may not even be depicted in the reference image, such 
as when the reference image was recorded at a time before the target was at that 
location in the scene area. By means of the spatial coordinates associated with each 
pixel in the reference image, the spatial scene coordinates of the unreferenced target 
may be discovered. In addition, by showing the corresponding location of the target 
point as mapped to the reference image, an observer examining the sensor image and 
its selected target point, and the reference image and its corresponding mapped target 
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point, can perform a judgment of the validity of the registration result, and of the target 
point placement in the reference image. 

[0024] Further features and advantages of the present invontion disclosure , as well as 

the structure and operation of various embodiments of the present invention disclosure , 

are described in detail below with reference to the accompanying drawings. 

[0025] FIG. 1 is a block diagram of a proforrod an embodiment of the processing 

architecture of the i nv e ntion disclosure for automatic image registration. 

[0026] FIG. 2 is a diagram illustrating a sensor footprint derivation in accordance with a 

pr o forr o d embodiment of the inv e ntion disclosure . 

[0027] FIG. 3 is a diagram illustrating a bounding box for a sensor footprint in 
accordance with a pr e f e rr e d embodiment of the i nv e nt i on disclosure . 

[0029] FIG. 5 illustrates an example of an image registration process in accordance With 
a pr e f e rr e d embodiment of the i nv e nt i on disclosure . 

[0030] Generally, in accordance with the present invontion disclosure , a small sensor 
image is matched to a larger reference image. The large reference image typically 
covers a relatively large area of the earth at a resolution of approximately the same, or 
better than, that normally expected to be seen in the sensor image. The reference area 
may be any area that can be the subject of a controlled imaging process that produces 
an image with known geometric characteristics and know geometric relationships 
between locations in the image and locations in the subject area. For example, the 
reference area may be a portion of a space assembly or an area on the human body. 
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This reference typically involves hundreds of thousands, or even millions or more of 
pixels (picture elements) in each of its two dimensions, and may comprise a pair of such 
images in a stereoscopic configuration that admits stereography in viewing and 
measurement. The reference image is geocoded so that a geographic location can be 
accurately associated with each pixel in the image, including an elevation if a stereo pair 
of images is used. For other types of reference areas, locations other than geographic 
are used as suited to the application, but some reference coordinate system is the basis 
for the location measurements. 

[0031] The sensor image, on the other hand, is fairly small, typically involving a few 
hundred or thousand pixels in each of its two dimensions. Resolution of the sensor 
image usually depends on the position of the sensor relative to the scene being imaged, 
but the relative positions of sensor and scene are normally restricted to provide some 
minimal desired resolution sufficient to observe appropriate detail in the scene and 
comparable to the detail shown in the reference image or stereo image pair. The sensor 
image typically depicts a different perspective from that of the reference image, often at 
a much lower, oblique, angle to the scene, whereas the reference image is typically 
from high overhead angles. On the other hand, the perspectives may be similar, such 
as for a synthetic aperture radar sensor, which typically presents a generally overhead 
view of the scene it images. These differences in geometry, whether arising from 
perspective differences or differences in sensor geometry, are a problem source 
addressed and solved by this i nv o ntion disclosure . 



Serial No. 10/817,476 



Page 7 of 27 



[0034] Numerous techniques of photogrammetry have been developed to identify 
acquisition parameters of sensors that produce characteristic perspective and scale 
properties in images. This i nv e nt i on disclosure makes use of such knowledge as is 
available about the images to reduce the matching problem to a tractable size so that a 
best match can be obtained along with a quality measure of the match to indicate its 
validity/invalidity. 

[0035] In accordance with a proforrod embodiment of the i nvontion disclosure , first the 
size of the reference image area to be searched is limited. With knowledge of the 
location of the sensor, its imaging properties (such as field of view and scale), and the 
location of the scene being sensed (such as the scene center), it is possible to 
determine the area within the reference image imaged by the sensor. This footprint of 
sensed image is extended by adding to it uncertainties in the locations of the sensor 
and scene. These uncertainties may include uncertainty as to look angles to the scene, 
range to the scene center, field of view, and pixel resolution in the scene. It is preferred 
to ensure that all uncertainties that influence the location of the sensed area within the 
reference image be taken into account. If the obliquity of the sensed image is low, so 
that a shallow view of the scene area is obtained by the sensor, it is possible that the 
area sensed will be quite large in the reference image. In this case, the scene area 
identified preferably is reduced to include amounts of area in front of and behind the 
scene center, as seen by the sensor, equal to a distance in front or behind the scene 
area of no more than twice the width of the sensed area, as seen by the sensor. 
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[0037] The chip is then distorted or warped to conform to the known geometry of the 
sensor image. In accordance with the inv e ntion disclosure , this involves several 
operations which may be performed in a variety of different sequences, or as a variety 
of combined operations, all of which result in a similar warping. One such sequence of 
operations will be described, but it is to be understood that other such operations know 
to those skilled in the art of image processing fall within the scope of this 
inv e nt i on disclosure . 

[0047] A preferred embodiment of the inv o ntion disclosure will further be described with 
reference to the drawings. Particularly with reference to FIG. 1, there is shown a block 
diagram of a processing architecture 10 for automatic image registration in accordance 
with a preferr e d embodiment of the i nv e ntion d isclosu re . Generally, the process 
comprises the following operations: 

[0071] 4b. If the reference image 28 consists of a left and right stereo pair, a chip is 
extracted from each to cover the AOI. The associated stereo model is then exploited to 
derive a DEM over the AOI. This DEM is accurately associated or aligned with each of 
the left and right chips, just as a reference DEM is associated or aligned with the 
reference image 28. Such stereo DEM extraction is performed using standard 
techniques in any number of commercially available software packages and well 
documented in the literature. It is the utilization of such techniques for automatic, 
unaided stereo extraction that is unique to the present i nv o ntion disclosure . 
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[0073] 5a. If the reference chip 30 is not an orthographic image, or is not close to 
orthographic, so that it exhibits perspective distortion (say more than ten degrees off 
from a perpendicular view of the scene area so that there is perspective distortion to be 
seen), it is desirable to remove the perspective distortion by producing the orthographic 
reference chip 48. This is accomplished using the reference chip 30 together with the 
reference DEM chip 42, as well as information about the reference image perspective. 
Such information is normally expressed in the form of mathematical mappings that 
transform coordinates of the reference scene area (such as geographic coordinates 
when the scene is of the ground and a height coordinate from the corresponding DEM) 
into coordinates of the digital or film image. The stereo extraction method of 
constructing a DEM also yields such information. Construction of the orthographic 
reference image chip 48 uses standard commercially available techniques. It is the 
utilization of such techniques to automatically produce orthographic images in an 
unaided fashion that is unique to the present i nv e nt i on disclosure . 

[0099] 8. Image match 60 is then carried out, between the synthetic perspective 
reference chip 58 and the sensor image 12. Again, there are many techniques that can 
be used, from a simple normalized image correlation, such as may be performed in the 
Fourier image transform domain, to a more robust, cross-spectral method like the 
Boeing General Pattern Match mutual information algorithm described in U.S. Pat. Nos. 
5,809,171; 5,890,808; 5,982,930; or 5,982,945 to another more robust, cross-spectral 
method like a mutual information algorithm described in P. Viola and W. Wells, 
"Alignment by Maximization of Mutual Information" International Conference on 
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Computer Vision, Boston, Mass., 1995. It is unique to the present i nv e nt i on disclosure 
that the only remaining difference between the two images after the processing 
described above, is a translation offset. This makes the match problem much easier to 
solve, requiring less computation and yielding a more accurate match result. 

[0103] FIG. 5 illustrates an example of an image registration process 100 of the present 
i nvention disclosure . 



[0113] While the present inv e ntion disclosure has been described by reference to 
specific embodiments and specific uses, it should be understood that other 
configurations and arrangements could be constructed, and different uses could be 
made, without departing from the scope of the i nv o ntio n disclosure as set forth in the 
following claims. 
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